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NASA aeronautics research has made decades of contributions to aviation. Both aircraft 
and air traffic management (ATM) systems in use today contain NASA-developed and NASA- 
sponsored technologies that improve safety and efficiency. Recent innovations in robotics and 
autonomy for automobiles and unmanned systems point to a future with increased personal 
mobility and access to transportation, including aviation. Automation and autonomous 
operations will transform the way we move people and goods. Achieving this mobility will 
require safe, robust, reliable operations for both the vehicle and the airspace and challenges 
to this inevitable future are being addressed now in government labs, universities, and 
industry. These challenges are the focus of NASA Langley Research Center’s Autonomy 
Incubator whose R&D portfolio includes mission planning, trajectory and path planning, 
object detection and avoidance, object classification, sensor fusion, controls, machine learning, 
computer vision, human-machine teaming, geo-containment, open architecture design and 
development, as well as the test and evaluation environment that will be critical to prove 
system reliability and support certification. Safe autonomous operations will be enabled via 
onboard sensing and perception systems in both data-rich and data-deprived environments. 
Applied autonomy will enable safety, efficiency and unprecedented mobility as people and 
goods take to the skies tomorrow just as we do on the road today. 
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CERTAIN 

City Environment for Range Testing of Autonomous Integrated Navigation 

CODE 

Cooperative Operations in Denied Environments 

DARPA 

Defense Advanced Research Projects Agency 

DDS 

Data Distribution Service 

DSB 

Defense Science Board 

FPV 

First Person View 

HITL 

Hardware-In-The-Loop 

HSI 

Human System Interaction 

ISS 

International Space Station 

LaRC 

Zangley Research Center 

MAAP 

Mid- Atlantic Aviation Partnership 

ONR 

Office of Naval Research 

OODA 

Observe Orient Decide Act 

PI 

Principle Investigator 

R&D 

Research & Development 

ROS 

Robot Operating System 

SA 

Situation Awareness 

UAS 

Unmanned Aerial System 

VTOL 

Vertical Take Off and Landing 


I. Introduction 

T HE Autonomy Incubator (AI) at NASA Langley Research Center (LaRC) was established in the spring of 2014 
to prepare the center workforce to meet the autonomy challenges that are anticipated in science, space exploration, 
and aeronautics as the NASA mission directorates look to enable new missions such as asteroid retrieval, planetary 
exploration, atmospheric sensing in historically inaccessible areas, and the integration of Unmanned Aerial Systems 
(UAS) into our everyday lives - all missions of increasing complexity, distance, proximity, pace, and/or accessibility. 
Building on decades of experience and success in the design, fabrication, and integration of safe and reliable automated 
systems for space and aeronautics, the LaRC Autonomy Incubator seeks to bridge the chasm between automation and 
autonomy and build systems that are capable of 

1 . sensing and perceiving their environments 

2. assessing their state 

3. making decisions in the face of uncertainty and with incomplete information 

4. acting on those decisions and 

5. learning from that experience. 

These autonomous systems will be non-deterministic and adaptive in much the same way that humans are. These 
systems (and/or the agents that comprise them) will earn trust (or lose it) much like humans. If successful, these 
intelligent agents will be able to transfer knowledge learned in one context to another (e.g., like a pilot transfer 
knowledge from one aircraft to another or a driver from one automobile to another) and acquire knowledge from other 
agents in the system. 

The Autonomy Incubator is a co-located team composed of researchers from multiple areas of expertise including 
but not limited to computer science, robotics, electrical engineering, mechanical engineering, aerospace engineering, 
psychology, machine vision, and machine learning. These Principle Investigators (Pis) are supported by qualified 
UAS pilots, flight safety personnel, range safety officers, and technicians. The team works together in an Agile work 
environment towards a common technical challenge around autonomous operations across NASA mission domains. 

II. Automation vs. Autonomy 

The distinction between automation and autonomy is not universally agreed upon. There are many models 
available for consideration, each with its own valid perspective and emphasis. Some assert that there is a continuum 
between human control with no assistance from the computer (human does it all) and computer decides everything 
(ignoring the human) 1 . This idea of “ignoring the human” implies that there is a human in the loop who might be 
observing but has no ability to intervene in the mission. Even though the definitions were created decades apart, this 
is consistent with the ICAO definition that an autonomous aircraft is “an unmanned aircraft that does not allow pilot 
intervention in the management of the flight” 2 . Other models base the distinction between automation and autonomy 
on complexity, again with a continuum between automation and autonomy, but based on whether the computer is 
acting as an expert (behaving expertly in a limited area of performance) vs. a decision support tool (helping the human 
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perform better perform his/her responsibilities). NASA’s Spacecraft Mission and Assessment and Replanning Tool 
(SMART) 3 is based on the OODA loop and divides each category (Observe, Orient, Decide, Act) into eight levels 
based on (as with [1]) function allocation between the human and the computer. The Department of Defense (DoD) 
Defense Science Board (DSB) asserts that the DoD should abandon the use of autonomy scales altogether and embrace 
a three-facet autonomous systems framework composed of cognitive echelon, mission timelines, and human-machine 
system trade spaces 4 . In this assertion, the DSB eliminates the point of confusion about automation and autonomy. 
Any scale that presents a continuum implies, whether intended or not, that the end state of “autonomy” can be achieved 
by increasing “automation” until some tipping point is reached, whether that continuum is based on human 
involvement or machine complexity or the approach to decision-making. 

While there may be some truth to the idea that sufficiently advanced automation is indistinguishable from 
autonomy (appropriated from Arthur C. Clarke’s Three Laws 5 ) especially when viewed from “the outside” or from 
the perspective of human-system interaction (HSI), it is what’s happening “under the hood” and, consequently, how 
these systems respond to new situations that distinguishes automation from autonomy. From the perspective of 
machine capabilities, the transition from automation to autonomy is a discrete step - a chasm 6 . In the transition from 
automation to autonomy, we move from systems that take predetermined action in (typically) structured and static 
environments to systems in dynamic and unstructured environments that sense and reason about their perceived state, 
assess that state for situation awareness (SA), and take action based on that SA. An autonomous system (or agent in 
that system) should be able to make decisions with incomplete or uncertain information or in the face of a new 
situation. This means that an intelligent agent in an autonomous system must be able to transfer knowledge from one 
situation to another and/or seek “advice” from other agents in the system and that truly autonomous systems will learn 
- from their own experiences and from each other. They will be non-deterministic and behaviors will emerge as the 
subsystem evolves over time. More simply stated 7 : 

The distinction between automation and autonomy is essentially the difference between relegation — assigning discrete, 
easily perfonned tasks — and delegation — assigning a given set of mission parameters... it’s the difference between 
machine-based execution and machine-based decision-making. 

In short, an autonomous system “has the bridge” or “the conn” when the commander or supervisor is not in or on the 
loop. Therefore, while the “Captain”, who is responsible for defining the mission, may have stepped away (or is so 
remote that s/he cannot tactically command), an autonomous system must have the authority and ability to redirect 
itself within some set of overarching mission goals based on its perceptions and processing without needing the explicit 
consent or direction of a human operator or user 8 . 

III. We Are Not Alone 

The Autonomy Incubator is one of many organizations researching and developing technologies and capabilities 
while working towards autonomous operations and including every effort at every university, company, and 
government agency that is rising to the autonomy challenge would be impossible. The following are a handful of 
research efforts that NASA is either leading or has direct involvement with either through cooperative research or via 
subject matter expertise. NASA’s Robonaut 9 is currently onboard the International Space Station (ISS) working 
alongside human crewmembers (Figure 1). The Defense Advanced Research Projects Agency (DARPA) Transformer 
TX program 10 initially aimed to build a “ground vehicle that is capable of configuring into a VTOL air vehicle with a 
maximum payload capability of approximately 1,000 lbs.” and has transitioned to Aerial Reconfigurable Embedded 
System (ARES) in Phase II, a VTOL flight module capable of carrying interchangeable mission-specific modules 
ranging from cargo to CASEVAC. The DARPA Aircrew Labor In-Cockpit Automation System (ALIAS) program 
envisions a “tailorable, drop-in, removable [robotic] kit that would enable high levels of automation in existing aircraft 
and facilitate reduced need for onboard crew” 11 . The ALIAS program also contains a Knowledge Acquisition 
component that speaks to machines learning to pilot an aircraft in a way similar to human learning. The DARPA 
Collaborative Operations in Denied Environment (CODE) program is focused on collaborative autonomy in denied 
or contested airspace and aims to create a capability such that “unmanned vehicles would continuously evaluate 
themselves and their environment and present recommendations for UAV team actions” to a single human mission 
supervisor” 12 for review and consent. ONR’s Autonomous Aerial Cargo Utility System (AACUS) 13 is an Innovative 
Naval Prototype (INP) that has demonstrated the feasibility of fully autonomous cargo delivery by an unmanned rotary 
wing vehicle capable of sensing and selecting a safe landing site in an area specified by a soldier in the field using a 
handheld tablet. In an effort similar to AACUS but focused on lunar landing, NASA’s Autonomous Landing and 
Hazard Avoidance Technology (ALHAT) 14 has demonstrated the ability to navigate and determine safe landing sites 
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on planetary surfaces (Figure 1). A suite of sensors and associated algorithms create an elevation map of the landing 
site to identify the location of hazards such as rock piles and craters and provide range and velocity data along with 
altitude measurements to help the vehicle locate the surface and land safely. 



Figure 1: Robonaut 2 and ALHAT (image credit NASA) 


IV. Our Mission 

In addition to recommending the abandonment of autonomy levels, the DSB 4 identified autonomy “gaps” and 
many of the AI’s ongoing R&D efforts are focused on these gap areas. 

• Most use of learning for autonomous navigation has been applied to ground vehicles and robots. One 
important area for future development in adaptive navigation is to refine the existing learning methods for 
effective use in alternative domains such as air and marine vehicles. 

• Most UxVs are required to operate in environments that are both unstructured and dynamic, where existing 
maps provide little guidance. Developing learning methods that can cope with such complex environments is 
an important challenge. 

The AI is exploring both traditional solutions as well as machine-learning based-approaches in these gap areas. 

The mission of NASA LaRC’s Autonomy Incubator is to rise to the challenge of data-degraded/deprived 
navigation in dynamic and unstructured environments. This is a an R&D challenge that must be met for NASA to 
retrieve an asteroid, explore planetary surfaces, measure pollution in historically inaccessible areas, and enable the 
integration of UAS into our National Airspace System (NAS) as well as our everyday lives. The AI has been focused 
on aerial navigation but these future NASA mission will require robotic solutions across space, air, sea, and land and 
we have begun broadening our portfolio to include multimodal navigation and payload delivery solutions. Further, we 
are exploring the efficacy of machine learning solutions in some aspects of this autonomous navigation problem space. 

A. Open Architecture 

A critical characteristic of our incubator effort is the ability to explore alternate methods and quickly integrate 
external solutions, and so we need a software architecture and framework that supports collaboration, interoperability, 
and scalability. We have designed and implemented an open architecture that employs Data Distribution Service 
(DDS) for Real-Time Systems for messaging middleware. DDS 15 is apublish-subscribe model (not unlike ROS, Robot 
Operating System 16 ) to take advantage of “plug-n-play” network topologies, portability between systems, Quality of 
Service (QoS) guarantees between software entities, and a abstracted interface that external entities can meet without 
becoming experts in DDS. Further, DDS moves away from the idea of a “core” process and the associated risk of a 
single point of failure to a distributed entity-to-entity middleware approach. 

The Autonomous Entity Operations Network (AEON) 17 is an open distributed software architecture and a highly 
configurable data fusion framework that provides plug-and-play compatibility with a wide array of computer systems, 
sensors, software, and controls hardware. It also supports a ground control system(s) that acts as a test-bed for 
integration of multi-modal robotic vehicles. 

B. Intelligent Flight Systems 
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The Autonomy Incubator falls under a broad area of innovation at NASA LaRC called Intelligent Flight Systems 
(IFS). IFS includes innovative vehicle design 18 , state-of-the-art wind tunnels 19 , cutting-edge sensors 20 , and advanced 
algorithms 21 . The AI focus is on algorithm development along with system design, integration, and test and utilizes 
procured vehicles and sensors from industry, academia, and other government organizations inside and outside of 
NASA. 


Autonomous mobility can be decomposed into a number of fundamental tasks. Assuming that a mission plan is 
available, an intelligent agent in the system must be able to: 

• Estimate its (ownship) state 

• Sense and perceive its world 

• Maintain and update its “map” of the world 

• Fuse information from multiple sensor sources 

• Detect and avoid objects/hazards 

• Classify objects as benign or hazardous 

• (Re)Plan its trajectory and path 

• Execute inner loop control for agile flight 

These tasks can and have been accomplished using traditional approaches - approaches applied successfully in ground 
robotics, adaptive control, etc. Within the AI, we are pursuing the advancement and integration of these more 
traditional (yet still groundbreaking) solutions as well as approaches that incorporate unsupervised, supervised, and 
reinforcement learning techniques. 

In the area of decision-making in critical phases of flight, we are currently applying supervised machine learning 
techniques to determining whether to “go around” during the automated landing approach of an unmanned fixed wing 
experimental testbed aircraft 22 . Initial tests of the go around decision capability focused on the transition from 
hardware-in-the-loop (HITL) simulation to actual flight tests (Figure 2) using a simple altitude threshold for the go 
around criteria. Additional relevant criteria such as centerline offset, glideslope error, estimated fuel remaining, as 
well as gust-induced transients are being incorporated into the learning model. The overall go around decision has 
been decomposed into lower level components to investigate the applicability of both supervised and unsupervised 
machine learning techniques. 



Figure 2: Automatic Go-Around Initiated at 200' AGL Due to Glide Slope Error 


A mission of particular interest to NASA is data collection in remote, cluttered environments such as under the 
tree canopy (e.g., rain forest) for Earth science. By applying reinforcement learning techniques to input from human 
operators, not unlike first person view (FPV) drone racing, the efficacy of a data set aggregation for this mission is 
being assessed 23 . Using video from a single forward-facing camera, computer vision based features relating to edge 
and gradient information are extracted and correlated with commands from a human operator to teach the autonomous 
system how to navigate around trees and other natural objects in an unmapped and cluttered forest environment. The 
software-in-the-loop simulation (SILSIM) and indoor flight test setup are shown in Figure 3. 
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Figure 3: Software-in-the-Loop Simulation and Indoor flight test of an “Under the Tree Canopy ” Mission 


An effective flight through a forest or in a disaster area requires a compelling pace of operations in terms of 
speed and agility. In support of these demanding missions, the AEON Flight Control System (AEON-FCS) is being 
developed. The AEON-FCS 24 is capable of implementing both classic guidance, navigation, and control (GNC) law 
algorithms in a centralized system architecture as well as a fully distributed control system, like AEON. This will 
allow a system design to break away from a centralized approach and place an AEON processor and inertial sensor 
set located close to the control effector(s), extending the distributed programming paradigm deep into the inner-loop. 
This distributed programming paradigm may also enable a data-centric approach to control laws such that a 
controller may be capable of being event driven rather than time synchronized enabling the performance gains 
required to realize challenging agile missions like under the canopy and surveillance and/or package delivery under 
the tree canopy or in dynamic unstructured environments such as disaster areas. 

There exists a large body of work in the area of object detection (and avoidance) utilizing any number of sensor 
modalities and algorithmic approaches. The Autonomy Incubator is pursuing ensemble learning and recognition 
system solutions for classifying an object once it has been detected. Target identification/classification, for at least 
the distinction between benign and hazardous objects or landing zones, is a critical functionality in autonomous 
navigation. A Bayesian inference approach 25 can be used to construct an initial set of known objects (training), 
distinguish between similar yet different objects in real-time, and update the systems belief space with respect to the 
world map. This classification is possible even in the presence of conflicting information from independent methods 
or algorithms contributing to the system-wide sensing and perception capability. 

Safe, reliable autonomous operations liberate the human operator from the duties of manual flight control or even 
the required knowledge of a pilot Instead, the human operator will likely be a scientist or a member of the general 
public who will set high-level supervisory goals and expect the system to execute the prescribed mission. In support 
of mission planning, the AI is exploring human-machine teaming approaches that employ natural language and gesture 
recognition 26 rather than a mouse and keyboard. With this ability of the system to understand both speech and gestures, 
operators unfamiliar with vehicle dynamics will be able to easily plan, initiate, and modify missions using interaction 
techniques that are familiar to them. This will foster better teaming between the operator and the autonomous agent 
which will help lower workload, increase situation awareness, and improve performance of the system as a whole. 

C. Test & Evaluation 

Access to relevant test environments is a challenge for aerial autonomous systems, whether airborne or spaceborne. 
Restrictions on autonomous and Beyond Line of Sight (BLOS) flight limit UAV access to airspace. Efforts such as 
the Mid-Atlantic Aviation Partnership (MAAP) 27 and the recent uptick in FAA 333 Exemptions are a step in the right 
direction but do not yet provide the instantaneous access to flight operational areas that are needed in the Al’s Agile 
approach to R&D. To that end, LaRC has initiated City Environment for Range Testing of Autonomous Integrated 
Navigation (CERTAIN). CERTAIN is a portfolio of complex environments (Figure 4) within which the challenges of 
UAS can be mastered. Customers can move from an indoor flight area of over 70,000 cubic feet in the Langley 
Autonomy and Robotics Center (home of the AI) to two types of contained outdoor flight (tethered and caged) and 
finally to outdoor flight in NASA LaRC’s onsite Phase I COA area. The NASA LaRC campus allows for overflight 
of urban landmarks, suburban buildings, and forested areas. The AI recently demonstrated an indoor GPS emulation 28 
capability that enables vehicles to fly seamlessly between indoor and outdoor environments without loss of signal. 


6 

American Institute of Aeronautics and Astronautics 



Figure 4: NASA LaRC's Operational Flight Areas 


V. Conclusions 

NASA LaRC’s Autonomy Incubator is focused on the technologies and capabilities that will be required to achieve 
autonomous missions in the domains of space exploration, aeronautics, and science. Building on decades of experience 
and success in the design, fabrication, and integration of safe and reliable automated systems for space and aeronautics, 
we are working towards a verifiable safe software architecture that supports rapid integration of software and hardware 
by leveraging community standards such as DDS and abstracting hardware-specific interfaces away from software 
system interfaces. Specific technologies and capabilities included in the AI R&D portfolio are object classification, 
agile control, human-machine teaming, machine learning, and decision-making in critical phases of flight. 
Additionally, the AI is guiding the first generation of an on-site T&E environment for performance assessment of 
autonomous aerial systems from both inside and outside of NASA. 
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